Image processing apparatus, imaging apparatus, movable object, and method for image processing

ABSTRACT

An image processing apparatus  10  includes an output unit  14  and a processor  16 . The output unit  14  outputs to a movable object  12  information indicating an action to be performed by the movable object  12 . The processor  16  determines a first action of the movable object  12  based on a state of a target, the state being sensed in imagery of a region around the movable object  12 . The processor  16  causes the output unit  14  to output information indicating the first action. The processor  16  determines a second action of the movable object  12  based on a state of the target, the state being sensed in imagery of the region around the movable object  12 , the imagery being captured after the first action of the movable object  12 . The processor  16  causes the output unit  14  to output information indicating the second action at a time depending on a speed of a predetermined motion of the target. The predetermined motion of the target is a motion that the target makes before the first action is determined or after the first action of the movable object  12.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Japanese Patent Application No.2019-117763 filed Jun. 25, 2019, the content of which is allincorporated herein by reference.

TECHNICAL FIELD

The present invention relates to an image processing apparatus, animaging apparatus, a movable object, and a method for image processing.

BACKGROUND ART

Technologies for sensing a target in imagery captured by using a camerahave been known in the art.

For example, a description has been provided with regard todetermination of moving speeds of a knee position and an upper-bodyposition of a pedestrian by using imagery captured by a vehicle-mountedcamera, which captures imagery of a region around a vehicle (refer toPTL 1).

CITATION LIST Patent Literature

PTL 1: Japanese Unexamined Patent Application Publication No.2010-066810

SUMMARY OF INVENTION

An image processing apparatus according to a first aspect of the presentdisclosure includes

an output unit that outputs to a movable object information indicatingan action to be performed by the movable object, and

a processor that determines a first action of the movable object basedon a state of a target, the state being sensed in captured imagery of aregion around the movable object, that causes the output unit to outputinformation indicating the first action, that determines a second actionof the movable object based on a state of the target, the state beingsensed in imagery of the region around the movable object, the imagerybeing captured after the first action of the movable object, and thatcauses the output unit to output information indicating the secondaction at a time depending on a speed of a predetermined motion that thetarget makes before the first action is determined or after the firstaction of the movable object.

An imaging apparatus according to a second aspect of the presentdisclosure includes

an image processing apparatus including an output unit that outputs to amovable object information indicating an action to be performed by themovable object, and a processor that determines a first action of themovable object based on a state of a target, the state being sensed incaptured imagery of a region around the movable object, that causes theoutput unit to output information indicating the first action, thatdetermines a second action of the movable object based on a state of thetarget, the state being sensed in imagery of the region around themovable object, the imagery being captured after the first action of themovable object, and that causes the output unit to output informationindicating the second action at a time depending on a speed of apredetermined motion that the target makes before the first action isdetermined or after the first action of the movable object, and

an imaging unit that generates the imagery of the region around themovable object.

A movable object according to a third aspect of the present disclosureincludes

an imaging apparatus including an image processing apparatus includingan output unit that outputs to the movable object information indicatingan action to be performed by the movable object and a processor thatdetermines a first action of the movable object based on a state of atarget, the state being sensed in captured imagery of a region aroundthe movable object, that causes the output unit to output informationindicating the first action, that determines a second action of themovable object based on a state of the target, the state being sensed inimagery of the region around the movable object, the imagery beingcaptured after the first action of the movable object, and that causesthe output unit to output information indicating the second action at atime depending on a speed of a predetermined motion that the targetmakes before the first action is determined or after the first action ofthe movable object, and an imaging unit that generates the imagery ofthe region around the movable object.

A method for image processing according to a fourth aspect of thepresent disclosure includes

sensing a state of a target in captured imagery of a region around amovable object,

determining a first action of the movable object based on the state ofthe target,

outputting information indicating the first action to the movableobject,

sensing a state of the target in imagery of the region around themovable object, the imagery being captured after the first action of themovable object,

determining a second action of the movable object based on the state ofthe target, and

outputting information indicating the second action to the movableobject at a time depending on a speed of a predetermined motion that thetarget makes before the first action is determined or after the firstaction of the movable object.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting an example of a configuration of animage processing apparatus according to an embodiment of the presentdisclosure.

FIG. 2 is an illustration depicting a first example in which a stalematebetween a movable object and a target occurs.

FIG. 3 is an illustration depicting a second example in which astalemate between a movable object and a target occurs.

FIG. 4 is a flowchart for describing a reaction process performed by acontroller depicted in FIG. 1.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a description will be provided with regard to an imageprocessing apparatus according to an embodiment of the present inventionwith reference to the drawings.

An imaging apparatus 11 including an image processing apparatus 10according to an embodiment of the present disclosure is mounted, forexample, in or on a movable object 12 as depicted in FIG. 1.

Examples of the movable object 12 may include a vehicle. Examples of avehicle may include an automobile, an industrial vehicle, a rail car, aresident car, and a fixed-wing aircraft that taxies along a runway.Examples of an automobile may include a passenger car, a truck, a bus, amotorcycle, and a trolleybus. Examples of an industrial vehicle mayinclude vehicles used for agriculture and vehicles used forconstruction. Examples of an industrial vehicle may include a forklifttruck and a golf cart. Examples of an industrial vehicle used foragriculture may include a tractor, a cultivator, a transplantingmachine, a binder, a combine, and a lawnmower. Examples of an industrialvehicle used for construction may include a bulldozer, a scraper, anexcavator, a crane truck, a dump truck, and a road roller. Examples of avehicle may include a human-powered vehicle. Categories of vehicles arenot limited to the examples described above. For example, examples of anautomobile may include an industrial vehicle that can run on a road. Onevehicle may be included in multiple categories.

The imaging apparatus 11 includes an imaging unit 13 and the imageprocessing apparatus 10.

Examples of the imaging unit 13 include a vehicle-mounted camera mountedin or on the movable object 12. The imaging unit 13 generates imagerythat captures a region around the movable object 12 and outputs theimagery to the image processing apparatus 10. Multiple imaging units 13may be mounted in or on the movable object 12. For example, if fourvehicle-mounted cameras are mounted in or on the movable object 12, themultiple imaging units 13 are disposed separately at a position fromwhich imagery of a neighboring region in front of the movable object 12can be captured, a position from which imagery of a neighboring regionbehind the movable object 1 can be captured, a position from whichimagery of a neighboring region on the left side of the movable object 1can be captured, and a position from which imagery of a neighboringregion on the right side of the movable object 12 can be captured.Imagery of the neighboring regions of the movable object 12 in alldirections can be captured with such an arrangement.

The imaging unit 13 at least includes imaging optics and an imagingelement. The imaging optics include optical components such as one ormore lenses and apertures. A lens included in the imaging optics has awide angle of view, and a fisheye lens or the like is an example. Theimaging optics form a subject image on a photosensitive surface of theimaging element. Examples of the imaging element include a chargecoupled device (CCD) image sensor and a complementary metal-oxidesemiconductor (CMOS) image sensor. The imaging element captures thesubject image formed on the photosensitive surface and generates imageryof a surrounding region.

The imaging unit 13 outputs the imagery of a surrounding regiongenerated by the imaging element to the image processing apparatus 10via wireline or wireless communication. The imaging unit 13 may outputthe imagery of the surrounding region to external devices such as anelectronic control unit (ECU), a display, and a navigation device, whichare mounted in or on the movable object 12.

The image processing apparatus 10 senses a state of a target by usingimagery of a region around the movable object 12 and determines inaccordance with the sensed state of the target an action to be performedby the movable object 12. Examples of a target include a person, anothermovable object, and an animal. Examples of a person as a target includea pedestrian and a person riding a bicycle. Examples of the movableobject 12 include a vehicle having an autonomous driving capability. Theterm “autonomous driving” used in the present embodiment indicatesautomation of all or part of operation to be performed by a user whodrives a vehicle. For example, autonomous driving may involve one ofLevel 1 to Level 5 defined by the Society of Automotive Engineers (SAE).In the following description, it is assumed that the movable object 12has a full autonomous driving capability in a level equal to or higherthan Level 4 defined by SAE. In the following description, a vehiclehaving a full autonomous driving capability is sometimes referred to asa full autonomous driving vehicle.

The image processing apparatus 10 includes an output unit 14, a storageunit 15, and a processor 16.

The output unit 14 is, for example, a communication interface andcommunicates with various ECUs of the movable object 12 via wireline orwireless communication. The output unit 14 outputs to the movable object12 information indicating an action to be performed by the movableobject 12. The output unit 14 outputs the information indicating theaction to be performed by the movable object 12, for example, bycommunicating with an ECU that controls driving of the movable object 12and an ECU that controls operations of the movable object 12, such ascausing a light and a turn signal to blink and sounding a horn.

The output unit 14, which is the communication interface, communicateswith various ECUs of the movable object 12 and may acquire informationfrom the various ECUs, the information including actions that have beenperformed by the movable object 12, a path ahead of the movable object12, and traffic in the surrounding region. The actions that have beenperformed by the movable object 12 include moving forward, movingbackward, stopping, slowing down, changing direction, switching on alight, flashing, causing a turn signal to blink, sounding a horn,presenting a display viewable from outside, maintaining the currentstate, outputting a voice message, and projecting a predetermined image.

The storage unit 15 includes one or more memories. The term “memory”used in the present embodiment indicates a semiconductor memory, amagnetic memory, or an optical memory as non-limiting examples. Eachmemory included in the storage unit 15 may function, for example, as amain storage device, an auxiliary storage device, or a cache memory. Thestorage unit 15 may store any information to be used for operations ofthe image processing apparatus 10. The storage unit 38 may store, forexample, a system program, an application program, and other programs.

The processor 16 includes at least one of a general-purpose processorand a dedicated processor. The general-purpose processor loads aspecific program and executes a specific function, and the dedicatedprocessor is designed exclusively for a specific process. The dedicatedprocessor may include an application specific integrated circuit (ASIC).The processor 16 may include a programmable logic device (PLD). The PLDmay include a field-programmable gate array (FPGA). The processor 16controls operation of each building component of the image processingapparatus 10.

The processor 16 determines whether a target is present, for example, inimagery of the surrounding region acquired from the imaging unit 13.Examples of a target include a flagman in a specific situation, apedestrian in a specific situation, and other movable objects. Examplesof a flagman 17 in a specific situation include a person who regulatestraffic when a path ahead of the movable object 12 is under constructionand one of two lanes is blocked as depicted in FIG. 2. Examples of apedestrian 18 in a specific situation include a person near an end of acrosswalk 19 across a path ahead of the movable object 12 as depicted inFIG. 3.

Once the processor 16 senses the presence of a target in imagery of thesurrounding region, the processor 16 senses a state of the target. Forexample, if the target is the flagman 17 or the pedestrian 18 in aspecific situation, the processor 16 senses a state in accordance withat least one piece of information regarding the flagman 17 or thepedestrian 18. Examples of one piece of information regarding theflagman 17 or the pedestrian 18 include the direction of movement, thefacing direction of the head or the direction of the line of sight, thefacial expression, and the limbs.

For example, if the target is the flagman 17 in the specific situation,which is depicted in FIG. 2, the state that is sensed is a state inwhich the flagman 17 is making either a gesture to allow passage, agesture to forbid passage, a gesture to give a direction to follow, oran unclear gesture. For example, if the target is the pedestrian 18 inthe specific situation, which is depicted in FIG. 3, examples of thestate that is sensed include a state in which the pedestrian 18 isstanding still in front of the crosswalk 19, a state in which thepedestrian 18 starts crossing, and a state in which the pedestrian 18 ismaking either a gesture indicating no intention of walking on thecrosswalk 19 or a gesture to encourage forward movement of the movableobject 12.

If the target is a person, such as the flagman 17 or the pedestrian 18,the processor 16 may sense a change in a facial expression by usingmultiple pieces of imagery of the surrounding region that formconsecutive frames. For example, an intensity of a specific feeling suchas anger may be scored with respect to a normal facial expression, and achange in a facial expression may be sensed as a change in the score.

Upon sensing the state of the target, the processor 16 determines basedon the state of the target a first action to be performed by the movableobject 12, the first action being selected from various actions that themovable object 12 can perform. Examples of the various actions that themovable object 12 can perform include moving forward, moving backward,stopping, slowing down, changing direction, switching on a light,flashing, causing a turn signal to blink, sounding a horn, presenting adisplay viewable from outside, maintaining the current state, outputtinga voice message, and projecting a predetermined image.

The processor 16 determines stopping as the first action if the targetis in one of the specific states that are defined for the target.Examples of the specific states include a state in which the flagman 17is making an unclear gesture in the specific situation and a state inwhich the pedestrian 18 is standing still in the specific situation.Once the first action is determined, the processor 16 causes the outputunit 14 to output information indicating the first action to variousECUs.

After outputting the first action or recognizing the notification sentfrom an ECU that has detected the execution of the first action, theprocessor 16 senses in the imagery of the surrounding region, which isacquired from the imaging unit 13, the state of the target for which thefirst action has been determined.

The processor 16 calculates the speed of a predetermined motiondetermined for the state of the target for which the first action hasbeen determined. The speed of a predetermined motion that is made eitherbefore the first action is determined or after the first action of themovable object 12 is calculated. If the target is the flagman 17 in thespecific situation, examples of the predetermined motion include theunclear gesture described above. If the target is the pedestrian 18 inthe specific situation, examples of the predetermined motion include thepedestrian 18 running and walking. The processor 16 calculates the speedof the predetermined motion by using multiple pieces of imagery of thesurrounding region that form consecutive frames.

The processor 16 determines based on the sensed state of the target asecond action to be performed by the movable object 12, the secondaction being selected from various actions that the movable object 12can perform. If it is determined based on the sensed state of the targetthat no meaningful response to the first action is obtained, theprocessor 16 may determine an action to attract attention of the targetas the second action.

A meaningful response indicates a response by which the intention of thetarget is recognizable. If the target is the flagman 17 in the specificsituation, examples of a meaningful response include a gesture to allowpassage, a gesture to forbid passage, and a gesture to provide adirection to follow. If the target is the pedestrian 18 in the specificsituation, examples of a meaningful response include a state in whichthe pedestrian 18 starts crossing and a state in which the pedestrian 18makes either a gesture indicating no intention of walking on thecrosswalk 19 or a gesture to encourage forward movement of the movableobject 12.

Examples of the action to attract attention of the target includeflashing, moving forward by a small amount, sounding a horn, outputtinga voice message, and projecting a predetermined image. Flashingindicates switching on a headlight of the movable object 12 in an upwarddirection (high beam) for an instant, and moving forward by a smallamount indicates moving the movable object 12 forward by several tens ofcentimeters. The second action may be formed by one of these actions toattract attention of the target or by an appropriate combination ofthese actions.

To determine the second action, the processor 16 may use imagery of thesurrounding region captured during a time period that elapses after thefirst action is performed, the time period depending on the calculatedspeed of the predetermined motion. The time period may be determined,for example, so as to decrease as the speed of the predetermined motionincreases.

The processor 16 causes the output unit 14 to output informationindicating the second action, which has been determined, to the variousECUs at a time depending on the calculated speed of the predeterminedmotion. The time may be determined with respect to the time point atwhich the information indicating the first action is output. The timemay be advanced as the speed of the predetermined motion increases.

The processor 16 may change the time in accordance with not only thepredetermined motion but also the traffic around the movable object 12.The time may be advanced as the traffic around the movable object 12becomes busier. If the target is a person, the processor 16 may changethe time in accordance with not only the predetermined motion andfurther the traffic in the surrounding region but also a change in thefacial expression of the target, which has been sensed. The time may beadvanced as the change in the facial expression of the target increases.

After outputting the second action or recognizing the notification sentfrom an ECU that has detected the execution of the second action, theprocessor 16 senses in the imagery of the surrounding region, which isacquired from the imaging unit 13, the state of the target for which thesecond action has been determined.

Thereafter, the processor 16 repeats an operation of sensing the stateof the target in the imagery of the surrounding region, determining anaction to be performed by the movable object 12, and outputting theaction that has been determined. The operation is repeated until thetarget reaches a specific stage, or the operation is repeated apredetermined number of times. The operation is similar to the operationbeing conducted after the first action is performed and until theinformation indicating the second action is output. The specific stageis, for example, a state in which the target is making a meaningfulresponse. The predetermined number of times is, for example, threetimes.

When causing the movable object 12 to perform multiple actions after thefirst action, the processor 16 may determine a different action eachtime. When determining a different action for each of the multipleactions performed after the first action, the processor 16 may determinean action in accordance with priorities predetermined for each state ofthe target.

The processor 16 may change the predetermined priorities in accordancewith the environment around the movable object 12. For example, for anoperation at night, the processor 16 may assign a low priority to anaction that may cause loud noise, such as sounding a horn, and assign ahigh priority to an action that does not cause loud noise, such asswitching on a light, flashing, or causing a turn signal to blink.

The processor 16 may terminate full autonomous driving of the movableobject 12 if the movable object 12 has been caused to perform apredetermined number of actions before the target reaches the specificstage.

Next, a reaction process performed by the processor 16 in the presentembodiment will be described with reference to a flowchart in FIG. 4.The reaction process starts when full autonomous driving of the movableobject 12 starts.

In step S100, the processor 16 determines whether a target is sensed inimagery of the surrounding region that is acquired from the imaging unit13. If it is determined that a target is not sensed, the process returnsto step S100. If it is determined that a target is sensed, the processproceeds to step S101.

In step S101, the processor 16 determines whether the state of thetarget that is sensed in step S100 is a specific state. If it isdetermined that the state of the target is not a specific state, theprocess returns to step S100. If it is determined that the state of thetarget is a specific state, the process proceeds to step S102.

In step S102, the processor 16 determines a first action to be performedby the movable object 12. After the determination, the process proceedsto step S103.

In step S103, the processor 16 calculates the speed of a predeterminedmotion that is determined for the state of the target for which thefirst action has been determined in step S102. After the calculation ofthe speed, the process proceeds to step S104.

In step S104, the processor 16 determines based on the speed calculatedin step S103 both a time to output information indicating a secondaction and a time period during which imagery of the surrounding regionis captured to determine the second action. The processor 16 alsodetermines in accordance with the state of the target for which thefirst action has been determined in step S102 the number of actions tobe performed on the target. After the determination, the processproceeds to step S105.

In step S105, the processor 16 causes the output unit 14 to outputinformation indicating the first action, which has been determined, tothe various ECUs. After the output, the process proceeds to step S106.

In step S106, based on a state of the target that is sensed in imageryof the surrounding region captured after the first action has beenperformed by the movable object 12, the processor 16 determines whetherthe target has made a meaningful response. If it is determined that ameaningful response has been made, full autonomous driving is performedin accordance with the meaningful response, and the process returns tostep S100. If it is determined that a meaningful response has not beenmade, the process proceeds to step S107.

In step S107, the processor 16 determines whether the number of actionsthat have been determined after the first action is performed exceedsthe number determined in step S104. If the determination is negative,the process proceeds to step S108. If the determination is affirmative,the process proceeds to step S110.

In step S108, the processor 16 senses a state of the target by usingimagery of the surrounding region captured during the time period, whichelapses after the first action is performed, the time period beingdetermined in step S104. The processor 16 determines based on the sensedstate of the target an action that the movable object 12 is to perform.After the action is determined, the process proceeds to step S109.

In step S109, at the time determined in step S104, the processor 16causes the output unit 14 to output information indicating the action,which has been determined in step S108, to the various ECUs. After theoutput, the process returns to step S106.

In step S110, the processor 16 causes the output unit 14 to outputinformation indicating the termination of the full autonomous driving tothe various ECUs. After the output, the reaction process ends.

The image processing apparatus 10 according to the present embodiment,which is configured as described above, determines the first action ofthe movable object 12 based on the state of the target sensed in theimagery of the surrounding region and determines the second action ofthe movable object 12 based on the state of the target sensed in theimagery of the surrounding region captured after the first action of themovable object 12. Advantageous effects provided by the image processingapparatus 10, which is configured in this way, will be described below.

A full autonomous driving vehicle that has been conceived in the artsometimes causes a stalemate with a target.

A stalemate can occur, for example, in a situation in which the flagman17 is standing in a path ahead of a movable object, which is a fullautonomous driving vehicle that has been conceived in the art, asdepicted in FIG. 2. The flagman 17 generally makes either a gesture toallow the movable object to pass or a gesture to forbid the movableobject from passing. However, the movable object can fail to recognize agesture, for example, because the gesture by the flagman 17 is differentfrom a gesture that is usually made. The movable object is generallyexpected to prioritize safety and stop under such circumstances. If thegesture by the flagman 17 keeps unrecognizable after the stop, themovable object remains stopped. In this way, it is possible that astalemate between the movable object and the flagman 17 occurs andsmooth traffic is obstructed.

Further, a stalemate can occur, for example, in a situation in which thepedestrian 18 is standing near an end of the crosswalk 19 across a pathahead of a movable object, which is a full autonomous driving vehiclethat has been conceived in the art, as depicted in FIG. 3. Under suchcircumstances, the movable object stops in front of the crosswalk 19 toallow the pedestrian 18 to walk on the crosswalk 19. However, thepedestrian 18 does not necessarily walk on the crosswalk 19 and can bestanding still near the crosswalk 19. Under such circumstances, themovable object remains stopped in front of the crosswalk 19 because thepedestrian 18 is standing still near the end of the crosswalk 19 despitehaving no intention to walk on the crosswalk 19. In this way, it ispossible that a stalemate between the movable object and the pedestrian18 occurs and smooth traffic is obstructed.

In contrast, if a meaningful response cannot be obtained from the targetin a situation in which a stalemate such as described above can occur,the image processing apparatus 10, which is configured as describeabove, performs a further action on the target and can thereby sense theintention of the target by encouraging a new response from the target.Thus, the image processing apparatus 10 can achieve smooth traffic bydecreasing the probability of causing a stalemate and can consequentlyimprove the convenience of using the movable object 12 fortransportation.

Further, the image processing apparatus 10 according to the presentembodiment outputs information indicating the second action at a timedepending on the speed of the predetermined motion of the target.Generally, it is possible that each target expects that the secondaction is performed at a different time in response to the motion madeas a response to the first action. For example, if the target is aperson, such a person as is either impatient or in a hurry is likely toexpect a quick action as a response from the movable object 12 to themotion or the state of the target. Further, in general, an impatientperson and a person being in a hurry quickly perform motions such as agesture, walking, and running. In such circumstances, since the imageprocessing apparatus 10, which is configured as described above, reactsto a response from the target and outputs the second action to themovable object 12 at a time that the target strongly wants, a responsefrom the target can more smoothly be encouraged. Accordingly, the imageprocessing apparatus 10 can further improve the convenience of using themovable object 12 for transportation.

Further, to determine the second action, the image processing apparatus10 according to the present embodiment uses the imagery of thesurrounding region captured during a time period that elapses after thefirst action of the movable object 12, the time period depending on thespeed of the predetermined motion. Generally, it is also possible thateach target responds to the first action of the movable object 12 with adifferent response time, which is the time period that elapses beforethe target responds. For example, if the target is a person, such aperson as is either impatient or in a hurry is likely to respond to thefirst action instantly, and such a person as is either placid or not ina hurry is likely to respond to the first action slowly. In suchcircumstances, the image processing apparatus 10, which is configured asdescribed above, can determine the second action after waiting for thetarget to respond to the first action of the movable object 12.Accordingly, since the image processing apparatus 10 can comprehend theintention of the target accurately and improve the accuracy of thecomprehension, the image processing apparatus 10 can further improve theconvenience of using the movable object 12 for transportation.

Further, the image processing apparatus 10 according to the presentembodiment also changes the time to output information indicating thesecond action in accordance with the traffic around the movable object12. Generally, a requirement for the promptness of an action of themovable object 12 changes in accordance with the traffic. In suchcircumstances, since not only the speed of a motion of the target butalso the traffic is reflected, the image processing apparatus 10, whichis configured as described above, can further improve the convenience ofusing the movable object 12 for transportation.

Further, the image processing apparatus 10 according to the presentembodiment determines in accordance with predetermined prioritiesmultiple actions to be performed after the first action. Various actionsthat can be performed by the movable object 12 have differentcapabilities of alerting the target and different degrees ofinconvenience caused to the neighbors. In such circumstances, the imageprocessing apparatus 10, which is configured as described above,determines actions in order of the suitability for the situation and canimprove the capability of alerting the target while reducing theinconvenience caused to the neighbors.

Further, the image processing apparatus 10 according to the presentembodiment changes the predetermined priorities in accordance with thesurrounding environment. The capability of alerting the target and thedegree of inconvenience caused to the neighbors can change depending onthe space where the movable object 12 is located, a time period, andother parameters. Accordingly, the image processing apparatus 10, whichis configured as described above, can further improve the capability ofalerting the target while reducing the inconvenience caused to theneighbors.

The present invention has been described with reference to the drawingsand based on the examples, and it should be noted that those skilled inthe art can easily make various modifications and corrections based onthe present disclosure. Thus, it should be noted that thosemodifications and corrections do not depart from the scope of thepresent invention.

For example, although the image processing apparatus 10 is mounted in oron the movable object 12 in the present embodiment, the image processingapparatus 10 need not be mounted in or on the movable object 12. Forexample, the image processing apparatus 10 may be an apparatus such as aserver on a network. Then, by using communication with the server viathe network, imagery of the surrounding region may be acquired from themovable object 12, and the information indicating the first action andthe actions following the first action may be supplied to the movableobject 12.

A number of aspects of content of this disclosure are presented as aseries of operations performed by hardware such as a computer systemthat can execute program commands. Examples of the hardware such as acomputer system include programmable data processing apparatuses such asa general-purpose computer, a personal computer (PC), a dedicatedcomputer, a workstation, a personal communications system (PCS) or apersonal mobile communications system, a mobile (cellular) telephone, amobile telephone having a data processing function, a radio frequencyidentification (RFID) receiver, a game machine, an electronic notepad, alaptop computer, and a global positioning system (GPS) receiver. It isto be noted that various operations in each embodiment are performedeither by a dedicated circuit (such as discrete logic gatesinterconnected with each other to perform a specific function) thatimplements program commands (software) or by components such as logicblocks and program modules that are executed by one or more processors.Examples of the one or more processors, which execute components such aslogic blocks and program modules, include one or more microprocessors, acentral processing unit (CPU), an application specific integratedcircuit (ASIC), a digital signal processor (DSP), a programmable logicdevice (PLD), a field programmable gate array (FPGA), a processor, acontroller, a micro-controller, a microprocessor, an electronic device,other devices designed to be able to perform the functions describedherein, and/or a combination of the devices described herein. Theembodiment described herein is implemented by using, for example,hardware, software, firmware, middleware, microcode, or a combinationthereof. A command may be a program code or a code segment for executinga necessary task. A command can be stored in a non-transitorymachine-readable storage medium or other media. A code segment mayrepresent any combination of a procedure, a function, a subprogram, aprogram, a routine, a subroutine, a module, a software package, a classor a command, a data structure, and a program statement. A code segmenttransmits and/or receives information, data arguments, variables, orstored content to and/or from other code segments or hardware circuits,thereby connecting to the other code segments or the hardware circuits.

REFERENCE SIGNS LIST

-   -   10 image processing apparatus    -   11 imaging apparatus    -   12 movable object    -   13 imaging unit    -   14 output unit    -   15 storage unit    -   16 processor    -   17 flagman    -   18 pedestrian    -   19 crosswalk

1. An image processing apparatus comprising: an output unit that outputsto a movable object information indicating an action to be performed bythe movable object; and a processor that determines a first action ofthe movable object based on a state of a target, the state being sensedin captured imagery of a region around the movable object; causes theoutput unit to output information indicating the first action;determines a second action of the movable object based on a second stateof the target, the second state being sensed in imagery of the regionaround the movable object that is captured after the first action of themovable object; and causes the output unit to output informationindicating the second action at a time depending on a speed of apredetermined motion that the target makes before the first action isdetermined or after the first action of the movable object.
 2. The imageprocessing apparatus according to claim 1, wherein the imagery of theregion around the movable object used in determining the second actionis captured during a time period that elapses after the first action ofthe movable object, the time period depending on the speed of thepredetermined motion.
 3. The image processing apparatus according toclaim 1, wherein the time is advanced as the speed of the predeterminedmotion increases.
 4. The image processing apparatus according to claim1, wherein the processor changes the time in accordance with trafficaround the movable object.
 5. The image processing apparatus accordingto claim 4, wherein the time is advanced as the traffic around themovable object becomes congested.
 6. The image processing apparatusaccording to claim 1, wherein the processor determines the second actionwhen it is determined based on the second state of the target that nomeaningful response to the first action is obtained from the target. 7.The image processing apparatus according to claim 1, wherein theprocessor determines as the first action and the second action at leastone of moving forward, moving backward, stopping, slowing down, changingdirection, switching on a light, flashing, causing a turn signal toblink, sounding a horn, presenting a display viewable from outside,maintaining a current state, outputting a voice message, and/orprojecting a predetermined image.
 8. The image processing apparatusaccording to claim 1, wherein the processor determines in accordancewith predetermined priorities a plurality of actions to be performedafter the first action, the plurality of actions being selected frommoving forward, moving backward, stopping, slowing down, changingdirection, switching on a light, flashing, causing a turn signal toblink, sounding a horn, presenting a display viewable from outside,maintaining a current state, outputting a voice message, and projectinga predetermined image.
 9. The image processing apparatus according toclaim 8, wherein the processor changes the predetermined priorities inaccordance with a surrounding environment.
 10. An imaging apparatuscomprising: the image processing apparatus according to claim 1; and animaging unit that generates the imagery of the region around the movableobject.
 11. A movable object comprising the imaging apparatus accordingto claim
 10. 12. A method for image processing, comprising: sensing astate of a target in captured imagery of a region around a movableobject; determining a first action of the movable object based on thestate of the target; outputting information indicating the first actionto the movable object; sensing a second state of the target in imageryof the region around the movable object, the imagery being capturedafter the first action of the movable object; determining a secondaction of the movable object based on the second state of the target;and outputting information indicating the second action to the movableobject at a time depending on a speed of a predetermined motion that thetarget makes before the first action is determined or after the firstaction of the movable object.